All you need is a good init

Layer-sequential unit-variance (LSUV) initialization – a simple method for weightinitialization for deep net learning – is proposed. The method consists of the twosteps. First, pre-initialize weights of each convolution or inner-product layer withorthonormal matrices. Second, proceed from the first to the final layer, normaliz-ing the variance of the output of each layer to be equal to one.Experiment with different activation functions (maxout, ReLU-family, tanh) showthat the proposed initialization leads to learning of very deep nets that (i) producesnetworks with test accuracy better or equal to standard methods and (ii) is at leastas fast as the complex schemes proposed specifically for very deep nets such asFitNets (Romero et al. (2015)) and Highway (Srivastava et al. (2015)).Performance is evaluated on GoogLeNet, CaffeNet, FitNets and Residual nets andthe state-of-the-art, or very close to it, is achieved on the MNIST, CIFAR-10/100and ImageNet datasets.